Method and device for determining a roadway state by means of a vehicle camera system

ABSTRACT

A method and a device are for determining the presence and type of a roadway coating by a vehicle camera system. The vehicle camera system records at least one image of the surroundings of the subject vehicle. The at least one image is evaluated to detect indications of the presence of a roadway coating including a characteristic feature of the roadway coating that results as the roadway is traveled over by the subject vehicle or by another vehicle. The detected indications and characteristic feature are taken into consideration in the determination of the presence and type of a roadway coating. The result of the determination of the roadway coating, or a friction coefficient estimation derived therefrom, is output to a driver assistance function, a vehicle control function or as information to the driver.

FIELD OF THE INVENTION

The invention relates to a method and a device for determining a roadwaycoating by means of a vehicle camera system.

BACKGROUND INFORMATION

Driver assistance systems support the driver in his driving task and arethus helping to make road traffic safer in future and to reduce accidentfigures. Camera-based driver assistance systems hereby detect thesurroundings of a vehicle. Camera systems which are located behind thewindshield detect the area in front of the vehicle according to thedriver's visual perception. The functional scopes of such assistancesystems extend from intelligent headlamp control to detecting anddisplaying speed limits to warnings in the event of the vehicle failingto keep to its lane or an impending collision. In addition to camerasystems, radar sensors, lidar sensors and/or laser scanners help todetect other vehicles, unprotected road users such as e.g. pedestriansand cyclists and the infrastructure such as e.g. crash barriers andtraffic lights. This is therefore creating the requirement to depict theimmediate vehicle environment more and more accurately.

The degree of automation of motor vehicles will continue to rise infuture. As a result, the level of equipping of vehicles with sensortechnology will also increase very significantly. Highly or fullyautomated vehicles will be equipped with a plurality of differentsensors and will thus allow a 360° view, in particular via camerasystems.

An essential part of a driver's driving task is correctly assessing theroadway state and, thus, the available friction coefficient betweentires and the roadway, in order to then adapt his driving styleaccordingly. In future, highly and fully automated vehicles will takeover the task of driving at least in partial areas. To this end, it isessential for the roadway state to be correctly detected and assessed bythe system.

DE 10 2004 018 088 A1 shows a roadway detection system having atemperature sensor, an ultrasonic sensor and a camera. The temperature,roughness and image data (roadway data) obtained from the sensors isfiltered and compared with reference data, and a level of security isgenerated for the comparison. The state of the roadway surface isdetected on the basis of the comparison of the filtered roadway datawith the reference data. The roadway surface (e.g. concrete, asphalt,dirt, grass, sand or gravel) and the state thereof (e.g. dry, icy,snowy, wet) can be classified in this way.

WO 2012/110030 A2 shows a method and a device for estimatingcoefficients of friction using a 3D camera, e.g. a stereo camera. Atleast one image of the environment of the vehicle is recorded by meansof the 3D camera. A height profile of the road surface is created in theentire area in front of the vehicle from the image data of the 3Dcamera. The anticipated local coefficient of friction of the roadsurface in the area in front of the vehicle is estimated from the heightprofile. In individual cases, the roadway surface can be classified,e.g. as a blanket of snow or a muddy dirt track, from special detectedheight profiles.

WO 2013/117186 A1 shows a method and a device for detecting thecondition of a roadway surface by means of a 3D camera. By means of the3D camera, at least one image of the surroundings extending in front ofthe vehicle is acquired. Height profiles of the roadway surface whichextend transversely to the direction of motion of the vehicle aredetermined from the image data of the 3D camera along a plurality oflines. The condition of the roadway surface is detected from thedetermined height profiles. In addition to the determined heightprofiles, 2D image data from at least one monocular camera of the 3Dcamera is optionally evaluated, e.g. by means of a texture or patternanalysis, and is incorporated into the detection of the condition of theroadway surface.

However, the known methods place high demands on the required sensortechnology. Therefore, in the case of the indicated methods and/ordevices, either a temperature and ultrasonic sensor are required inaddition to a camera, or the camera must be configured as a 3D sensor,so that the classification results are sufficiently robust.

SUMMARY OF THE INVENTION

An object of an embodiment of this invention is therefore to indicate aroadway state determination by means of a camera which, when differentvehicle camera system configurations are used, including, for example,just one mono camera system, makes possible an optimized roadway statedetection and friction coefficient estimation derived therefrom.

One of the main ideas of the invention is to specifically detectindications of the existence of a roadway coating as the roadway istraveled over by a vehicle by means of a camera system and imageprocessing. A roadway coating is e.g. snow, rainwater, ice, leaves,dust, i.e. media/objects which lie flat (blanket, carpet) on the roadwaysurface (asphalt, tar, concrete, etc.). The phenomenon of lying flat onthe roadway surface can be described as a blanket or carpet of themedium or of the objects, however the entire roadway surface does nothave to be covered. Different roadway coatings typically show differentbehaviors when they are traveled over by a vehicle. Some of them can beobserved and/or detected with a camera and recognized and/or identifiedin a subsequent evaluation of the image, from which conclusionsregarding the presence and the nature of the roadway coating can bedrawn.

A method for determining a roadway coating by means of a vehicle camerasystem according to the invention comprises the following steps: bymeans of the vehicle camera system, at least one image of thesurroundings of the vehicle is recorded. The at least one image isevaluated in order to detect indications of the presence of a roadwaycoating as the roadway is traveled over by the vehicle with the vehiclecamera system or by another vehicle. The detected indications are takeninto consideration in the determination of a roadway coating. The resultof the determination of the roadway coating can preferably be combinedwith a direct roadway state detection by means of a classification or anapplication of a neural network and output to a driver assistancefunction, to a vehicle control function or also as information to thedriver. Driver assistance functions can in particular comprise collisionwarnings, emergency braking applications or even emergency steering,whilst vehicle control functions can in particular comprise occupantprotection measures (airbag anticipatory control, brake anticipatorycontrol, belt tensioner anticipatory control) as well as (partially)autonomous braking and/or steering interventions. In a preferred use ofthe method according to the invention, the detected indications and/orthe result of the determination of the roadway coating are taken intoconsideration in the design of driver assistance functions and vehiclecontrol functions.

The result of the determination of the roadway coating can in particularbe incorporated into an estimate of the friction coefficient for theroadway area which is shown in the image, or a current frictioncoefficient which is otherwise detected or a predictively detectedfriction coefficient. This is because the roadway coating has asignificant influence on the actual friction coefficient. The frictioncoefficient, also known as the friction value, adhesion coefficient,(adhesive) friction coefficient or coefficient of friction indicates themaximum force with reference to the wheel load which can be transmittedbetween a roadway surface and a vehicle tire (e.g. in the tangentialdirection) and is thus an essential measure of driving safety. Inaddition to the roadway state, the properties of the tire are requiredin order to determine the friction value completely.

According to an advantageous embodiment, the effects of precipitation(for example rain, snow, hail or even fog) in the image, on the roadwayand on the vehicles or vehicle windshields are detected as indicationsduring the evaluation of the image.

The type and quantity of precipitation can be determined during theevaluation of the image. The detection direction of the camera systemcan hereby be advantageously linked with the current direction of motionof the vehicle. Thus, when heavy rainfall is detected by a front camera,it can be concluded during the evaluation of the image that the roadwayin front of the vehicle will be wet.

In a preferred embodiment, effects of a roadway coating as it istraveled over by at least one tire of a vehicle are detected asindications during the evaluation of the image. Effects are hereby inparticular modifications of the roadway surface including the roadwaycoating as it is traveled over by a vehicle or, more precisely, as it istraveled over by at least one tire of the vehicle.

At least one region in the at least one image is advantageouslydetermined, from which indications of the presence of a roadway coatingare to be inferred. These are in particular image regions which includea vehicle tire travelling over said roadway coating or which are locatedadjacent to (at the side of or behind) such a vehicle tire.

The region in the at least one image is preferably supplied to aclassifier which maps the detected indications onto a quantity ofclasses which are each assigned to one roadway coating.

According to an advantageous embodiment, the vehicle camera system has arear and/or lateral detection area of the surroundings of the hostvehicle. The host vehicle is hereby the vehicle which has the vehiclecamera system. In this way, indications of the presence of a roadwaycoating as the roadway is traveled over by the host vehicle can bedetected during the evaluation of the image. The evaluation of the imagecan, in the process, advantageously be limited to a fixed image region,in which effects of an existing roadway coating are typically shown,therefore e.g. a region behind a tire or at the back of the host vehiclewhich is shown by a rear camera or a region adjacent to a tire of thehost vehicle which is shown by a side camera.

The advantage with this embodiment is that, in order to determine aroadway coating, the driver is not dependent on other vehicles locatedin the detection area of the vehicle camera system.

In a preferred embodiment, the vehicle camera system has a detectionarea located in front of the host vehicle, so that indications of thepresence of a roadway coating as the roadway is traveled over by avehicle travelling in front, a vehicle travelling on a road whichintersects with the road on which the host vehicle is travelling or anoncoming vehicle are detected during the evaluation of the image.

To that end, the evaluation of the image may, in particular, be limitedto an image region in which the back and/or the side of a vehicletravelling in front or the vehicle front and/or the side of an oncomingvehicle is/are shown. The relevant image region can preferably bedetermined on the basis of the tire contact zones with the roadway inthe image.

One advantage of this embodiment is that a predictive determination of aroadway coating is possible.

The indications of the presence of a roadway coating as the roadway istraveled over advantageously comprise whirled-up components of theroadway coating. These include in particular the detection of spraywater or sprayed slush, spray mist or spray, snow turbulence, leaves aswell as sand or dust.

According to a preferred embodiment, the indications of the presence ofa roadway coating comprise obstructed views in the field of view of thevehicle camera system due to the roadway being traveled over by avehicle travelling in front, a vehicle travelling on a road whichintersects with the road on which the host vehicle is travelling or anoncoming vehicle.

The indications of the presence of a roadway coating as the roadway istraveled over by a vehicle preferably comprise tire ruts in the roadwaycoating. In the case of special roadway coatings, tire ruts are formeddirectly behind the tires of a moving vehicle, sometimes the impressionof the tire profile can even be detected within a tire rut e.g. in thecase of a coating of snow.

During the determination of the presence of a roadway coating, inaddition to the indications of the presence of a roadway coating as theroadway is traveled over by a vehicle, typical generic properties ofdifferent roadway coatings can, in addition, advantageously be detectedand taken into consideration. This means properties which arecharacteristic of roadway coatings and which can be recognized from acamera image even without a vehicle travelling over the roadway surface,e.g. puddles and reflections in the case of rainwater or the topology ofthe surface. In particular, headlights of oncoming vehicles arereflected more strongly on a rain-wet or icy roadway than on a dryroadway.

The classifier preferably comprises a neural network which has beentrained, in order to be able to assign the detected indications to aroadway coating. The neural network can in particular also continuallylearn to assign indications to a class of roadway coatings.

According to an advantageous further development of the invention, theresult of the determination of the roadway coating is combined with theresult of a roadway state detection or classification and the frictioncoefficient is estimated therefrom. In particular, the roadway statedetection or classification determines the material and geometricroadway surface e.g. rough or smooth tar, asphalt or concrete, possiblywith existing ruts, and can in particular be based on image data fromthe same vehicle camera system.

The invention additionally relates to a device for determining a roadwaycoating comprising a vehicle camera system, an evaluating unit, adetermining unit and an output unit. The vehicle camera system isconfigured to record at least one image of the surroundings of thevehicle. The evaluating unit is configured to evaluate the at least oneimage, in order to detect indications of the presence of a roadwaycoating by precipitation and/or as the roadway is traveled over by avehicle (E, F). The determining unit is configured to determine aroadway coating taking into consideration the detected indications. Theoutput unit is set up to output the result detected by the determiningunit. The evaluating, determining and output units can in particular bepart of the control unit of the vehicle camera system or of othervehicle control units.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be explained in more detail below by means of figuresand embodiment examples, where:

FIG. 1 schematically shows a binary camera image of a vehicle tire as arain-wet roadway is traveled over.

FIG. 2 shows different detection areas of a complex camera system of avehicle.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS OF THE INVENTION

FIG. 1 shows a black and white illustration of a camera image of avehicle tire traveling over a rain-wet roadway. The vehicle tire ismounted on a wheel rim (white with black outlines, circular) and thevehicle moves to the right. Water drops and streams of water (black dotsand streams) can be seen, which are displaced and accelerated as thetire travels over the roadway surface covered with water. Spray water isof course formed, starting from the vehicle tire, as wet roads aretraveled over. Due to the high surface pressure, the tire displaces thestanding water on the roadway on all sides. The phenomenon isparticularly pronounced with trucks, the tires of which have acorrespondingly higher surface pressure and must displace more waterthan cars. Looking in the direction of travel, the spray water is mainlylocated behind and laterally adjacent to the vehicle tire. It can bedetected there by a vehicle camera and recognized as spray water byimage processing, from which it can be concluded that a layer ofrainwater is coating the roadway.

If slush is present on the roadway, spraying slush is produced in acomparable way, which can be detected and recognized as such. Dependingon the roadway coating, e.g. in the case of a damp instead of a wetroadway, and depending on the structure of the roadway surface, no spraywater will be produced as the road coating is traveled over, insteadonly spray mist or spray or both will be produced. This can also bedetected by a vehicle camera and can be recognized by image processingas spray mist or spray.

When roads covered with new snow or powder snow are traveled over,typical snow turbulence occurs, which spreads to the sides of thevehicle and, as a light covering of snow, to the rear of the vehicle.Other roadway coatings such as e.g. a layer of leaves or a sandy surfaceon a paved roadway are also whirled up when they are traveled over andcan be detected and recognized with a vehicle camera.

In addition, in the case of some roadway coatings, additionalindications in the camera image can be detected and identified. Thus, aswet or snow-covered roadways are traveled over or as slush is alsotraveled over, tire ruts are formed behind a moving vehicle, sometimesthe impression of the tire profile within a tire rut can even bedetected e.g. when there is a thin layer of snow. In addition,precipitation in the form of rain, snow or hail, and even fog, can bedetected as indications of the roadway state in the camera image.

Different indications of the existence of a roadway coating detectedfrom camera images are therefore advantageously taken into considerationin the classification of the nature of the roadway coating.

The different classes can be coarsely divided (no roadwaycoating/water/snow), but can also be more finely divided (e.g. in thecase of snow: powder snow up to 5 mm high, powder snow >5 mm high, solidsnow, slush). This subdivision can in particular be made as a functionof an assignment of the class to an average friction coefficient, and/orthe change of the assigned average friction coefficient during thetransition from one subclass to another.

FIG. 2 shows the detection areas (1 a-1 f, 2, 3) of a camera systemarranged in or on a first vehicle (E). A second vehicle (F) is locatedlaterally offset in front of the first vehicle (E) in the direction oftravel. The camera system of the first vehicle (E) comprises threedifferent camera subsystems (1, 2 and 3): a panoramic view system (1)comprising six individual camera sensors with wide-angle detection areas(1 a-1 f) which, together, allow 360° detection of the vehicle, a frontcamera with a forward-facing detection area (2) and a rear camera with arear-facing detection area (3).

Indications of the presence of a roadway coating as the roadway istraveled over by vehicles travelling in front of the host vehicle (F) orby oncoming vehicles (not shown) can be detected with camera sensorswhich comprise a forward-facing detection area (1 a, 1 b, 2) in thedirection of travel.

This offers the advantage of a predictive determination of the roadwaycoating. A stereo camera can preferably be used as a front camera, as aresult of which a spatial resolution of the recorded image data isachieved. Objects such as e.g. spray water drops can be analyzed withrespect to their spatial form and extent, which improves theclassification of an existing roadway coating.

The image processing for determining a roadway coating can beadvantageously limited to one or more regions of the camera image, inwhich a tire contact zone of the roadway is located.

In the case of a vehicle travelling in front (F) an image region can beevaluated, in which the rear of the vehicle of the vehicle travelling infront is located.

Indications of the presence of a roadway coating as the roadway istraveled over by the tires of the host vehicle (E) can be detected withcamera sensors, which comprise a detection area (3, 1 e, 1 f, 1 c, 1 d)facing backwards or to the side in the direction of travel.

The advantage with this is that the driver is not dependent on othervehicles (F) travelling in front or offset but, thanks to the detectionand determination of the effects caused by the host vehicle (E) by meansof rear and/or laterally oriented sensor technology, can determine acurrently relevant roadway coating, independently of other vehicles (F).Since vehicles will in future be increasingly equipped with 360° camerasensor technology which records a surround view which can be displayedto the driver e.g. as a “top view” in a bird's eye view, the reliabledetermination of a roadway coating is realistic.

The invention claimed is:
 1. A method of determining a roadway coatingon a roadway with a vehicle camera system of a subject vehicle, whereinthe roadway coating is formed by extraneous media or objects lying onthe roadway, wherein plural different types of roadway coatingsrespectively exhibit different characteristic features when driven overby the subject vehicle or another vehicle, wherein the plural differenttypes of roadway coatings comprise snow, water, slush, ice, leaves, andsand, and wherein the method comprises: with the vehicle camera system,recording camera images of surroundings of the subject vehicle includingat least one camera image of the roadway coating on the roadway, whichexhibits an actual characteristic feature resulting from the roadwaycoating being driven over by the subject vehicle or the another vehicle;evaluating at least a partial region of the at least one camera image byimage processing to detect, from only the at least one camera image,indications of a presence of the roadway coating including the actualcharacteristic feature of the roadway coating as the roadway is drivenover by the subject vehicle or by the another vehicle; determining thepresence of the roadway coating on the roadway and making a conclusionabout a type of the roadway coating among the plural different types ofroadway coatings in consideration of and dependent on only theindications including the actual characteristic feature that weredetected from only the at least one camera image, and outputting aresult of the determining of the presence and the conclusion about thetype of the roadway coating.
 2. The method according to claim 1, whereinthe extraneous media which forms the roadway coating comprisesprecipitation lying on the roadway, and wherein the indications of thepresence of the roadway coating detected from only the at least onecamera image further comprise effects of the precipitation on theroadway, on the subject vehicle, on the another vehicle, or on a vehiclewindshield of the subject vehicle or of the another vehicle.
 3. Themethod according to claim 1, wherein the actual characteristic featurecomprises effects on the roadway coating as it is traveled over by atleast one tire of the subject vehicle or of the another vehicle, whicheffects are detected in the evaluating of the at least one camera image.4. The method according to claim 1, further comprising determining atleast one selected partial region in the at least one camera image, andlimiting the evaluating of the at least one camera image to only the atleast one selected partial region thereof so as to detect theindications of the presence of the roadway coating only from the atleast one selected partial region of the camera image.
 5. The methodaccording to claim 1, wherein the making of the conclusion is performedby a classifier which maps the detected actual characteristic featureonto a plurality of classes characterized by the differentcharacteristic features which are respectively assigned to the pluraldifferent types of roadway coatings.
 6. The method according to claim 1,wherein the vehicle camera system has a rear and/or lateral detectionarea of the surroundings located behind and/or beside the subjectvehicle so that the indications of the presence of the roadway coatingincluding the actual characteristic feature exhibited by the roadwaycoating as the roadway is traveled over by the subject vehicle aredetected in the evaluating of the at least one camera image.
 7. Themethod according to claim 1, wherein the vehicle camera system has aforward detection area of the surroundings located in front of thesubject vehicle so that the indications of the presence of the roadwaycoating including the actual characteristic feature exhibited by theroadway coating as the roadway is traveled over by the another vehicleare detected in the evaluating of the at least one camera image, whereinthe another vehicle comprises a leading vehicle traveling in front ofthe subject vehicle, an oncoming vehicle traveling opposite the subjectvehicle, or a crossing vehicle traveling on a roadway path thatintersects a roadway path of the roadway on which the subject vehicle istraveling.
 8. The method according to claim 1, wherein the actualcharacteristic feature of the roadway coating comprises whirled-upcomponents of the roadway coating which are detectable at least in thepartial region of the at least one camera image in the evaluatingthereof.
 9. The method according to claim 1, wherein the indications ofthe presence of the roadway coating detected from only the at least onecamera image further comprise obstructed views in a field of view of thevehicle camera system because the roadway is being traveled over by theanother vehicle, which comprises a leading vehicle traveling in front ofthe subject vehicle, an oncoming vehicle traveling opposite the subjectvehicle, or a crossing vehicle traveling on a roadway path thatintersects a roadway path of the roadway on which the subject vehicle istraveling.
 10. The method according to claim 1, wherein the actualcharacteristic feature of the roadway coating comprises tire ruts in theroadway coating which are detectable at least in the partial region ofthe at least one camera image in the evaluating thereof.
 11. The methodaccording to claim 5, wherein the classifier comprises a neural networkthat has been trained to be able to assign the detected actualcharacteristic feature to a specific matching one of the differentcharacteristic features of the plural different types of roadwaycoatings.
 12. The method according to claim 1, further comprisingperforming a roadway state detection or classification of the roadway,combining the result of the determining of the presence of the roadwaycoating with a result of the roadway state detection or classification,and estimating a friction coefficient of the roadway from the combinedresults.
 13. A device for determining a roadway coating on a roadway,wherein the roadway coating is formed by extraneous media or objectslying on the roadway, wherein plural different types of roadway coatingsrespectively exhibit different characteristic features when driven overby a subject vehicle or another vehicle, wherein the plural differenttypes of roadway coatings comprise snow, water, slush, ice, leaves, andsand, and wherein the device comprises: a vehicle camera system on thesubject vehicle, which vehicle camera system is configured to recordcamera images of surroundings of the subject vehicle including at leastone camera image of the roadway coating on the roadway, which exhibitsan actual characteristic feature resulting from the roadway coatingbeing driven over by the subject vehicle or the another vehicle, anevaluating unit which is configured to evaluate at least a partialregion of the at least one camera image by image processing to detect,from only the at least one camera image, indications of a presence ofthe roadway coating including the actual characteristic feature of theroadway coating as the roadway is driven over by the subject vehicle orby the another vehicle, a determining unit which is configured todetermine the presence of the roadway coating on the roadway and to makea conclusion about a type of the roadway coating among the pluraldifferent types of roadway coatings in consideration of and dependent ononly the indications including the actual characteristic feature thatwere detected from only the at least one camera image, and an outputunit which is configured to output a result of the determining of thepresence and the conclusion about the type of the roadway coating.
 14. Amethod for a driver assistance system of a subject vehicle, comprisingthe following steps performed by the driver assistance system: a) with acamera system of the subject vehicle recording surroundings of thesubject vehicle and producing corresponding camera image data includingat least one camera image of a roadway coating on a roadway, wherein theroadway coating is formed by extraneous media or objects lying on theroadway, and wherein the roadway coating exhibits a characteristicfeature resulting from being driven over by the subject vehicle oranother vehicle; b) evaluating at least a partial region of the at leastone camera image by image processing to detect, from only the at leastone camera image, indications of a presence of the roadway coatingincluding the characteristic feature of the roadway coating; c)classifying the characteristic feature, which has been detected fromonly the at least one camera image, among pre-determined pluraldifferent characteristic features that are respectively exhibited byplural different types of roadway coatings as a result of being drivenover, wherein the plural different types of roadway coatings comprisesnow, water, slush, ice, leaves, and sand; d) based on a result of theclassifying, making a conclusion about a type of the roadway coating onthe roadway among the plural different types of roadway coatings; and e)outputting the conclusion about the type of the roadway coating.
 15. Themethod according to claim 14, wherein the classifying of thecharacteristic feature and the making of the conclusion are based ononly the camera image data, without consideration of temperature sensordata and without consideration of ultrasonic sensor data.
 16. The methodaccording to claim 14, wherein the plural different characteristicfeatures comprise whirled-up or sprayed components of the extraneousmedia or objects that form the roadway coating, which whirled-up orsprayed components are whirled-up or sprayed into the air above theroadway when the roadway coating is driven over by the vehicle, whereinthe whirled-up or sprayed components are selected from the groupconsisting of water spray, slush spray, snow spray, spray mist,whirled-up water, whirled-up snow, sprayed snow, whirled-up leaves,sprayed leaves, whirled-up sand, sprayed sand, whirled-up dust, andsprayed dust.
 17. The method according to claim 14, wherein the camerasystem comprises a camera with a detection zone of the camera directedto a rear area behind the subject vehicle or a lateral area to a side ofthe subject vehicle, wherein the characteristic feature exhibited by theroadway coating is a characteristic feature resulting from being drivenover by the subject vehicle, and wherein the plural differentcharacteristic features respectively exhibited by the plural differentroadway coatings are characteristic features resulting from being drivenover by the subject vehicle.